Stream4Flow: Real-time IP Flow Host Monitoring using Apache Spark

In this paper, we present Stream4Flow, a framework for cyber situational awareness based on Apache Spark Streaming. We demonstrate utilization of Stream4Flow for real-time IP flow host monitoring in a large campus network. Contemporary IP flow analysis systems are not designed for the continuous host monitoring. Gaining the detailed overview of each host is not straightforward with these systems due to connection-based paradigm and performance challenges. We show that distributed stream processing is a natural solution for detailed IP flow host monitoring. Moreover, we describe a real-time host monitoring workflow in data streams in detail and present advantages of flow-based host monitoring in Apache Spark including real-time host profiling, dynamic level of detail and granularity

On the Impact of Flow Monitoring Configuration

Flow monitoring has become an essential source of information for intrusion detection systems and various forms of network data analytics. However, the attention of researchers is focused primarily on the utilisation of the flow data, and the process of flow data creation is often neglected. This lack of consideration negatively affects the results of data analytics. Either the results are suboptimal due to the low quality of the flow data, or a description of the configuration of the flow monitoring system is missing, which leads to irreproducible results. The goal of this paper is to demonstrate how the configuration of the flow monitoring system affects the resulting data. The most basic flow monitoring configuration variables are the flow expiration timeouts. We analyse their effect on the number of created flow records to show their importance. Moreover, we demonstrate that the choice of the flow expiration timeouts can have a severe impact on the network data analytics. The use-case of Slowloris attack detection is used as an example to illustrate this fact

IT Operations Analytics: Root Cause Analysis via Complex Event Processing

IT operation analytics (ITOA) is used for discovering complex patterns in data from IT systems. The analytics process still includes a significant portion of human interaction which makes the analysis costly and error-prone. Human operators need to formulate queries over the collected data to identify the complex patterns. Since the queries describe complex relations, the queries are usually multilevel, perplexing, and complicated to create. For the querying the complex relations, complex event processing methods are successfully used in other domains. In this paper, we demonstrate an application of the complex event processing principles in the ITOA domain. We adjust T-Rex complex event processing engine and improve TESLA event processing language to suit for ITOA tasks. Our demonstration includes two real-world use-cases. We show the utilization of the complex event processing for root cause analysis and demonstrate the natural formulation of complex queries that results in the reduction of the volume of the required human interaction

Cyber Situation Awareness via IP Flow Monitoring

Cyber situation awareness has been recognized as a vital requirement for effective cyber defense. Cyber situation awareness allows cybersecurity operators to identify, understand, and anticipate incoming threats. Achieving and maintaining the cyber situation awareness is a challenging task given the continuous evolution of the computer networks, increasing volume and speeds of the data in a network, and rising number of threats to network security. Our work contributes to the continuous evolution of cyber situation awareness by the research of novel approaches to the perception and comprehension of a computer network. We concentrate our research efforts on the domain of IP flow network monitoring. We propose improvements to the IP flow monitoring techniques that enable the enhanced perception of a computer network. Further, we conduct detailed analyses of network traffic, which allows for an in-depth understanding of host behavior in a computer network. Last but not least, we propose a novel approach to IP flow network monitoring that enables real-time cyber situation awareness

Real-time Analysis of NetFlow Data for Generating Network Traffic Statistics using Apache Spark

Abstract—In this paper, we present a framework for the realtime generation of network traffic statistics on Apache Spark Streaming, a modern distributed stream processing system. Our previous results showed that stream processing systems provide enough throughput to process a large volume of NetFlow data and hence they are suitable for network traffic monitoring. This paper describes the integration of Apache Spark Streaming into a current network monitoring architecture. We prove that it is possible to implement the same basic methods for NetFlow data analysis in the stream processing framework as in the traditional ones. Moreover, our stream processing implementation discovers new information which is not available when using traditional network monitoring approaches

On Information Value of Top N Statistics

In the era of Internet of Things (IoT), the volume of the monitored data from IoT network is enormous. However, not all data provide sufficient or relevant information. Since the analysis of big data is both resource and time exhausting, only relevant information should be analysed. In this paper, we scrutinize the widely used Top N statistics and evaluate its information value with respect to gathering information about individual hosts in the network. All theoretical discussions are evaluated on the real-world data. Moreover, we provide an assessment of statistic's suitability for identifying a host in network traffic. The results of the paper should assist data analyst of IoT network data

Network Traffic Characterisation Using Flow-Based Statistics

Performing research on live network traffic requires the traffic to be well documented and described. The results of such research are heavily dependent on the particular network. This paper presents a study of network characteristics, which can be used to describe the behaviour of a network. We propose a number of characteristics that can be collected from the networks and evaluate them on five different networks of Masaryk University. The proposed characteristics cover IP, transport and application layers of the network traffic. Moreover, they reflect strong day-night and weekday patterns that are present in most of the networks. Variation in the characteristics between the networks indicates that they can be used for the description and differentiation of the networks. Furthermore, a weak correlation between the chosen characteristics implies their independence and contribution to network description

Identifying Operating System Using Flow-based Traffic Fingerprinting

Many vulnerabilities are operating system specific. Information about the OS of all hosts in a network represents a valuable asset for network administrators. While OS detection in small networks is an easy task, expanding the same process on a large scale becomes a challenge. The weak performance, high speed traffic and large amount of hosts for OS detection are issues to overcome. In this paper we propose a flow based framework for large scale OS detection. Furthermore, we describe the framework implementation into a flow probe, provide performance comparison and share remarks on deployment in a real world network

Toward Real-time Network-wide Cyber Situational Awareness

In today's complex computer networks, we are constantly facing a risk of data loss, system compromise, or intellectual property theft. The complexity of the networks hinders their effective defense. A Network-wide Cyber Situational Awareness (NwCSA) has been introduced to assist a network security administrator with network security. The concept, however, faces several challenges that hinder an efficient application of the NwCSA in a real-world environment. The challenges include the overload of raw data, low speed of reaction, and a lack of context and unified view on a network. In this paper, we present a novel framework that faces above mentioned challenges. The framework leverages a distributed data stream processing system and methods for real-time big data processing. The framework is evaluated with respect to stated requirements on systems for NwCSA. Moreover, we present a prototype framework implementation and provide lessons learned from its real-world deployment

Cloud-based Security Research Testbed: A DDoS Use Case

In this paper we present a cloud-based research testbed designed to aid network security managers. The testbed enables operators to emulate various network topologies, services, and to analyze attacks threatening these systems. A possibility to test results of network management measures is desired, since testing these measures in a production environment is always not possible. We demonstrate a testbed use case, which aids to scrutinize network behavior under attack. Our use case is based on a large DDoS attack which targeted network infrastructure and web servers in Czech Republic in March, 2013.Článek popisuje cloudové testovací prostředí navržené pro využití správci síťové bezpečnosti. Testovací prostředí umožňuje simulovat různé síťové topologie, služby a provádět analýzu útoků zaměřených proti provozovaným službám. Protože není vždy možné provádět tuto analýzu na reálné síti, je potřeba mít k dipozici testovací prostředí s výše uvedenými vlastnostmi. V tomto článku je popsána případová studie využití cloudového testovacího k simulaci a analýze chování sítě pod náporem DDoS útoku. Studie je inspirována DDoS útokem na významné síťové infrastruktury v březnu 2013